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FOREWORD 


The  model  presented  in  this  paper  was  developed  under  the  Training 
and  Education  Project  in  the  Engagement  Simulation  Technical  Area  of 
the  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) . 
The  goal  of  this  project  was  to  provide  quantitative  methods  for  evalu¬ 
ating  unit  proficiency.  The  means  for  achieving  this  goal  included 
basic  research  in  criterion-referenced  test  methodology,  measurement 
and  scaling  models,  and  decisionmaking  implications  of  test  score 
interpretat ion . 

The  Combat  Operations  Training  Effectiveness  Analysis  Model  (COTEAM) 
is  a  criterion-referenced  evaluation  system  based  on  the  Army  Training 
and  Evaluation  Program  (ARTEP) .  The  model  integrates  aspects  of  all 
parts  of  the  Technical  Area  work  program:  evaluation  of  small  combat 
units  in  a  simulated  combat  environment  (ARTEP,  REALTRAIN) ,  ARTEP  evalu¬ 
ator  training,  methods  for  developing  performance  criteria,  and  improved 
training-diagnostic  feedback  procedures.  Research  from  all  of  these 
areas  is  related  to  the  COTEAM  framework. 

Anticipated  future  research  under  the  Training  and  Education  Proj¬ 
ect  includes  the  development  of  a  computer  model  for  performance  evalua¬ 
tion  and  development  of  measurement,  scaling,  scoring,  decisionmaking, 
and  quality-control  models  for  use  in  performance  evaluations  when 
criterion-referenced  testing  procedures  are  employed. 

ARI  research  in  this  area  is  conducted  as  an  in-house  research 
effort.  The  research  program  is  responsive  to  the  requirements  of  Army 
Project  2Q762722A764 . 
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COMBAT  OPERATIONS  TRAINING  EFFECTIVENESS  ANALYSIS  MODEL: 
1979  PERSPECTIVE 


BRIEF 


Requirement : 

To  present  the  current  status  of  theory  and  research  relevant  to 
the  Combat  Operations  Training  Effectiveness  Analysis  Model  (COTEAM) , 
an  evaluation  system  for  small  combat  units. 


Description  of  the  Model: 

The  current  Army  Training  and  Evaluation  Program,  ARTEP,  possesses 
several  weaknesses.  COTEAM  is  based  on  the  ARTEP  framework  but  cor¬ 
rects  its  shortcomings  by  providing  (a)  a  realistic  simulated  combat 
environment  in  which  units  perform  tactical  operations  and  objective 
data  can  be  obtained,  (b)  procedures  for  defining  standards  against 
which  unit  performance  can  be  compared,  and  (c)  techniques  for  assess¬ 
ing  training  deficiencies  and  levels  of  training  or  combat  readiness. 


Findings : 

A  review  of  the  current  status  of  the  COTEAM  evaluation  system  in¬ 
dicates  that  research  is  being  conducted  on  all  aspects  of  the  model: 

(a)  The  ARTEP  manual  is  being  modified  to  provide  empirically  determined 
behavioral  objectives,  a  list  of  training  diagnostic  behaviors,  objec¬ 
tive  standards,  and  a  program  of  instruction  for  ARTEP  evaluators. 

(b)  The  COTEAM  simulated  combat  environment  exercises  are  superior  to 
ARTEP  field  exercises  in  promoting  tactical  proficiency,  terrain  appreci¬ 
ation,  and  use  of  cover  and  concealment;  in  providing  immediate,  valid 
feedback  to  individuals  and  weapon  systems;  in  enhancing  troop  motiva¬ 
tion;  and  in  allowing  ample  opportunity  for  collection  of  objective  data 
on  unit  performance.  (c)  A  set  of  systematic  methods  for  defining  spe¬ 
cific  performance  criteria  against  which  unit  performance  can  be  compared 
are  being  developed,  validated,  and  implemented.  (d)  A  comparison  proc¬ 
ess,  in  which  behaviors  observed  in  field  exercises  are  compared  to  per¬ 
formance  criteria,  is  being  explored.  This  Comparison  subsystem  allows 
decisions  to  be  made  concerning  the  training  level  and  combat  readiness 
of  the  unit  being  evaluated. 
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COMBAT  OPERATIONS  TRAINING  EFFECTIVENESS 
ANALYSIS  MODEL:  1979  PERSPECTIVE 


INTRODUCTION 

In  recent  years,  the  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  has  been  involved  in  a  systematic  program  of  re¬ 
search  on  unit  evaluation.  The  goal  of  the  research  is  to  develop  a 
criterion-referenced  system  for  evaluation  of  unit  tactical  performance. 
Considerable  progress  has  been  made  in  developing  evaluation  systems 
that  provide  (a)  a  realistic  simulated  combat  environment  in  which  units 
perform  tactical  operations  and  in  which  objective  performance  data  can 
be  obtained,  (b)  procedures  for  defining  standards  against  which  unit 
performance  can  be  compared,  and  (c)  techniques  for  assessing  training 
deficiencies  and  training/combat  level  readiness.  One  of  the  current 
models,  the  Combat  Operations  Training  Effectiveness  Analysis  Model 
(COTEAM) ,  uses  the  current  Army  evaluation  system — the  Army  Training  and 
Evaluation  Program  (ARTEP) — as  a  starting  point  but  also  includes  the 
aforementioned  critical  features. 1 

As  currently  used,  the  ARTEP  includes  an  ARTEP  manual,  a  field 
exercise  evaluation,  and  a  training  program  based  on  the  evaluation. 

Figure  1  presents  a  flow  chart  of  the  system.  As  the  figure  suggests, 
the  ARTEP  manual  drives  the  evaluation  system.  Based  on  doctrine  and 
military  expertise,  the  ARTEP  manual  was  designed  to  be  a  training  and 
evaluation  guide.  Evaluation  requirements  are  stated  in  terms  of  spe¬ 
cific  unit  performance  objectives  and  focus  on  whether  a  unit  can  per¬ 
form  specified  missions.  Task  statements  are  written  at  an  integrated, 
functional,  mission-oriented  level.  Conditions  for  performing  each 
task  have  been  specified  to  allow  greater  standardization  of  the  evalu¬ 
ation.  Training  and  evaluation  standards  provide  a  relatively  objective 
basis  on  which  the  evaluator  can  judge  the  unit's  performance  on  a  par¬ 
ticular  task;  the  standards  also  can  be  used  to  develop  training  programs. 

The  external  conditions  are  factors  that  influence  the  ARTEP  field 
exercise  and  that  can  be  controlled  by  the  evaluator.  The  ARTEP  manual 
and  doctrine  determine  mission,  posture,  personnel,  weapons,  ammunition, 
and  petroleum,  oils,  and  lubricants  (POL) .  The  evaluation  team  deter¬ 
mines  whether  the  terrain  and  weather  are  suitable,  and  the  time  of  day 


1This  model  was  originally  outlined  by  Eugene  Johnson,  K.  I.  Epstein, 
and  Angelo  Mirabella  as  part  of  ARI  basic  research  on  criterion- 
referenced  evaluation.  Its  features  have  been  described  in  Epstein  end 
Johnson,  1976,  and  Mirabella,  1977.  Related  evaluation  concepts  have 
been  discussed  in  Erwin,  1976;  Root,  Knerr,  Severino,  and  Word,  1978; 
Sulzen  (undated  draft  report) ,  Sulzen  and  Root,  1976;  and  Sulzen,  Root, 
and  Epstein,  1976. 
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Figure  1.  ARTEP  evaluation  system 


and  intelligence  for  the  exercise.  (These  are  examples  of  external 
conditions,  not  an  exhaustive  list.) 


After  considering  the  external  conditions,  the  evaluation  team 
designs  a  field  exercise  based  on  the  unit  performance  objectives  in 
the  ARTEP  manual .  Many  characteristics  of  the  unit  being  evaluated 
affect  its  performance  in  the  field  exercise;  for  example,  the  indi¬ 
vidual  skills  of  the  unit's  members,  the  unit's  training  level,  and  the 
unit's  leadership.  During  the  field  exercise,  the  evaluation  team  uses 
the  ARTEP  manual  as  a  guide  in  the  evaluation  process.  Based  on  the 
unit's  performance  in  the  field  exercise,  the  evaluation  team  makes  de¬ 
cisions  on  training  level/combat  readiness  and  training  remediation. 

A  training  program  then  is  designed  to  strengthen  the  unit's  deficien¬ 
cies  and  maintain  its  proficiencies.  As  Figure  1  shows,  this  training 
program  primarily  affects  the  characteristics  of  tne  unit,  which  in 
turn  affect  the  unit's  performance  in  its  next  field  exercise. 

The  ARTEP  has  several  major  weaknesses  that  make  an  improved  com¬ 
bat  evaluation  system  necessary.  The  first  weakness  lies  in  the  lack 
of  standardized  or  scientific  procedures  for  determining  the  tasks, 
subtasks,  and  standards  in  the  Training  and  Evaluation  Outlines  (T&EOs) 
of  the  ARTEP  manuals.  The  current  content  of  the  T&EOs  was  chosen  by 
military  experts  without  the  benefit  of  available  procedures  for  in¬ 
suring  consensus  among  different  teams  of  experts. 

The  second  major  weakness  in  current  ARTEP  use  is  that  the  field 
exercises  are  often  unrealistic  and  do  not  provide  objective  data  for 
the  evaluation  team.  If  the  field  exercises  do  not  provide  a  realistic 
combat  environment,  then  the  evaluation  of  combat  units  in  this  setting 
has  little  or  no  validity. 

Third,  the  ARTEP  manual  provides  little  or  no  guidance  to  evalu¬ 
ators  on  how  to  design  exercises,  measure  unit  performance,  determine 
training  proficiencies/deficiencies,  or  evaluate  the  observed  perform¬ 
ance.  Effective  and  efficient  measurement  methodology  is  lacking,  par¬ 
ticularly  with  regard  to  objective  measures  of  performance  and  optimal 
utilization  of  limited  evaluation  personnel  and  equipment.  Current 
practice  relies  heavily  on  subjective  evaluations  and  the  ability  of 
the  evaluation  team  to  anticipate  critical  events  during  an  exercise 
and  to  insure  that  those  events  are  observed.  At  best,  such  evaluation 
yields  good  qualitative  judgments,  but  there  is  no  assurance  that  the 
judgments  will  not  be  more  or  less  arbitrary. 

The  ARTEP  is  currently  dominated  by  unsystematized,  unaided,  idio¬ 
syncratic  human  judgments.  The  standards  tend  to  be  open  to  consider¬ 
able  interpretation.  The  complexities  of  combat  are  too  great  to  be 
handled  by  an  evaluation  system  based  solely  on  an  intuitive,  subjective, 
ascientific  approach.  Although  human  judgment  must  play  a  significant 
role  in  the  evaluation  process,  it  must  be  systematized,  aided  by 
information-processing  technology,  and  made  less  idiosyncratic.  The 
lack  of  objective  data  and  an  explicit  criterion  data  base  make  it 


difficult  to  interpret  the  results  of  an  evaluation,  to  compare  differ¬ 
ent  units,  or  to  compare  the  same  unit  at  different  times  or  locations. 
The  evaluation  system  needs  to  be  driven  by  objective,  quantitative 
data.  Systematized,  aided  human  judgment  should  be  supported  by  quan¬ 
titative  measures  as  much  as  possible.  Because  of  the  complex  nature 
of  field  exercises,  the  system  should  incorporate  multiple  measures  at 
several  levels  of  resolution. 

The  fourth  weakness  in  current  ARTEP  use  is  the  lack  of  guidance 
on  how  users  are  to  deal  with  the  partially  stochastic  nature  of  com¬ 
bat  .  Estimates  must  be  made  of  the  degree  to  which  chance  events  re¬ 
duce  the  accuracy  of  effectiveness  measures  and  therefore  reduce  the 
validity  of  the  evaluation  decisions.  Low  empirical  correlations  be¬ 
tween  variables  that  would  be  expected  to  be  highly  related  on  logical 
grounds  can  occur  for  many  reasons.  Such  reasons  include  stochastic 
effects,  uniquely  effective  or  ineffective  behavior  by  the  opposing  team, 
nontraining  sources  of  variance  that  may  mask  learning  effects,  and  the 
suppressor  variable  effects  of  events  that  intervene  between  a  process 
and  a  later  outcome.  This  suggests  that  the  evaluation  system  should 
consider  the  processes  leading  up  to  the  final  outcomes,  as  well  as  the 
final  ou  t  c ome  s . 

Finally,  the  AKTEP  does  not  provide  guidance  in  how  to  develop 
training  programs  from  the  field  exercise  evaluation.  Although  the 
AKTEP  is  a  training  and  evaluation  guide,  units  typically  train  to  pre¬ 
pare  for  taking  an  ARTEP  evaluation,  or  test.  After  an  ARTEP  exercise, 
feedback  to  the  unit  is  provided,  but  the  feedback  is  neither  timely 
enough  nor  in  a  form  that  is  useful  for  developing  a  training  program. 
Furthermore,  many  aspects  of  lower  echelon  performance  are  neglected 
during  the  larger  ARTEP  field  exercises;  this  violates  the  ARTEP  concept 
of  multi-echelon  training  and  evaluation  and  prevents  the  integration  of 
pertinent  information  from  lower  echelons  into  the  training  program. 

Thus,  critical  aspects  of  training  within  the  ARTEP  are  lost  (Havron  & 
Wanschura,  1979). 


COTEAM 

The  COTEAM  evaluation  system  uses  the  ARTEP  as  a  starting  point 
and  systematically  addresses  the  ARTEP's  weaknesses.  Figure  2  presents 
a  flow  chart  of  COTEAM.  A  comparison  of  Figures  1  and  2  shows  that 
COTEAM  is  based  on  the  ARTEP  evaluation  system.  The  model  modifies  the 
current  ARTEP  manual  and  adds  these  main  components:  (a)  an  engagement 
simulation  field  exercise  test  bed,  (b)  the  expected  values  subsystem, 
and  (c)  a  comtvtrison  subsystem  for  training  diagnosis. 
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Figure  2.  COTEAM  evaluation  system 


Modifications  to  the  ARTEP  Manual 


Traditionally,  the  ARTEP  manual  and  the  front-end  analysis  of  be¬ 
havioral  objectives,  performance  variables,  and  measurements  have  re¬ 
lied  heavily  on  expert  military  judgments.  At  ARI,  efforts  have  been 
made  to  use  standardized,  scientific  procedures  based  on  empirical  data, 
as  a  way  to  increase  experts'  consensus  and  as  a  means  of  testing  and 
veri-fyi-n9  expert  opinion.  These  methods  also  provide  a  basis  for  re¬ 
ducing  many  expert  opinions  down  to  a  manageable  set  of  variables  that 
are  most  important  in  assessing  unit  performance  in  field  exercises. 

One  research  effort  has  focused  on  identifying  variables  that  ap¬ 
pear  to  be  useful  in  discriminating  among  units  differing  in  tactical 
proficiency.  Using  a  large  data  base  from  combined  arms  field  exercises, 
critical  incidents  within  exercises  have  been  identified,  as  well  as 
the  causes  or  precipitating  factors  of  particularly  damaging  or  helpful 
tactical  events.  These  critical  events  include  particularly  long  delays 
between  acquiring  targets  and  delivering  indirect  fire,  units  that  ex¬ 
pose  themselves  for’  long  periods  of  time,  and  breakdowns  in  communica¬ 
tions.  A  prototype  ARTEP  manual  is  being  developed  with  a  set  of  diag¬ 
nostics  based  on  these  critical  incidents.  These  diagnostics  are 
enabling  behaviors";  that  is,  a  unit  that  performs  the  behavior  is  more 
likely  to  complete  its  task  or  mission  than  a  unit  that  does  not.  Evalu¬ 
ators  also  can  use  these  diagnostics  to  help  develop  training  programs 
based  on  the  unit's  ARTEP  evaluation.  By  noting  which  tasks  were  not 
completed  successfully  and,  with  the  aid  of  the  diagnostics,  tracing 
back  to  determine  why  a  tack  was  not  completed,  the  evaluator  can  assess 
the  unit's  training  deficiencies  and  develop  a  training  program  to  cor¬ 
rect  them. 

A  second  research  effort  is  aimed  at  defining  the  major  dimensions 
or  factors  that  military  judges  use  to  assess  unit  performance  in  field 
exercises.  Multidimensional  scaling  (MDS)  techniques  are  being  explored 
as  a  means  of  identifying  these  important  factors.  Using  written  nar¬ 
ratives  of  15  different  armor /anti-armor  field  exercises,  military  offi¬ 
cers  were  asked  to  rate  each  unit's  performances.  One  of  the  15  narra¬ 
tives  was  used  as  a  "target";  the  other  narratives  were  compared  to  the 
target  along  certain  specific  criteria.  For  instance,  the  officers  were 
asked  to  'judge  how  similar  each  narrative  is  to  the  target  narrative 
with  respect  to  the  performance  of  the  combat  unit  in  the  narratives." 
Each  of  the  15  narratives  was  used  as  the  target;  comparisons  were  made 
between  all  the  other  narratives  and  the  target.  A  similarity  matrix 
composed  of  all  the  comparisons  was  used  as  input  to  an  MDS  computer 
program,  which  mathematically  determined  how 'many  factors,  or  dimensions, 
the  judges  used  in  their  comparisons.  The  results  yielded  a  three- 
dimensional  space;  that  is,  the  judges  used  only  three  factors  to  evalu¬ 
ate  unit  performance. 
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The  next  step  in  this  research  is  to  define  or  label  these  dimen¬ 
sions.  To  do  this,  a  list  of  possible  dimension-defining  attributes 
is  composed,  a  new  set  of  military  judges  then  ranks  the  narratives 
with  respect  to  how  much  each  narrative  is  characterized  by  the  attri¬ 
butes,  and  these  comparisons  are  used  as  input  to  a  second  MDS  analysis. 
The  set  of  three  attributes  that  defines  a  three-dimensional  space 
most  similar  to  the  original  is  the  set  of  factors  that  military  judges 
consider  to  be  important  in  assessing  unit  performance  in  field  exercises. 

A  third  area  in  which  the  current  ARTEP  is  being  improved  is  in 
the  training  given  to  the  evaluation  team.  A  detailed  program  of  in¬ 
struction  (POI)  (Havron,  McCullough,  McFarling,  &  Wanschura,  1979)  for 
evaluators  is  being  developed  and  field  tested.  The  POI  teaches  the 
evaluators  how  to  design  and  set  up  field  exercises,  obtain  observations, 
assess  unit  performance,  and  provide  feedback  to  the  units.  The  guide 
includes  the  rationale  for  the  development  of  the  POI,  which  stresses 
the  roles  and  importance  of  the  evaluators;  a  command  planning  module, 
which  presents  the  planning  necessary  to  conduct  an  evaluator  training 
session;  and  the  POI  itself. 

The  POI  provides  structured  guidance  for  the  evaluators  in  all  as¬ 
pects  of  their  duties,  including  each  evaluator's  role  and  the  required 
coordination  of  roles,  methods  for  controlling  field  exercises,  use  of 
pyrotechnic  simulators,  terrain  reconnaissance  for  exercise  lanes,  data 
analysis,  report  preparation,  and  postexercise  critiques.  The  POI  con¬ 
stitutes  the  first  significant  attempt  to  develop  guides  for  persons 
who  evaluate  unit  performance  in  field  exercises. 


Engagement  Simulation  Field  Exercise  Test  Bed 

Although  some  engagement  simulation  (ES)  technology  has  been  fielded 
by  the  Army,  most  current  ARTEP  field  exercises  do  not  use  objective 
methods  for  simulating  and  assessing  casualties.  Some  question  exists, 
therefore,  of  whether  the  behavior  displayed  in  these  exercises  repre¬ 
sents  true  combat  performance.  The  evolution  of  engagement  simulation 
from  SCOPES  and  REALTRAIN  (TC  71-5)  to  the  Multiple  Integrated  Laser 
Engagement  System  (MILES)  has  increased  dramatically  the  realism  of  the 
combat  environment  and  the  objectivity  of  casualty  assessment.  The 
National  Training  Center  (NTC)  will  move  this  evolution  a  step  farther 
by  providing  a  computer-controlled,  instrumented  range  with  unique  weapon 
signatures  for  each  weapon  system  in  field  exercises  up  to  battalion  size. 

Engagement  simulation  exercises  are  superior  to  the  ARTEP  field  ex¬ 
ercises  in  promoting  tactical  behaviors,  terrain  appreciation,  cover, 
and  concealment  (Havron,  McFarling,  Hill,  &  Wanschura,  1979b;  Scott, 

Meliza  et  al.,  1979;  Banks  et  al. ,  1977;  Meliza  et  al.,  1979;  Scott, 

Banks  et  al.,  1979).  Engagement  simulation  provides  immediate  and  valid 
feedback  to  individuals  and  weapon  crews  and  enhances  troop  motivation 
and  presumably  readiness  to  learn  (Sulzen  &  Bleda,  1979).  The  after 
action  review  (AAR)  gives  the  unit,  weapon  crews,  and  individual  soldiers 
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an  opportunity  to  learn  from  their  mistakes  and  benefit  from  the  experi¬ 
ences  of  the  other  participants. 

In  addition  to  providing  a  realistic  combat  environment,  ES  exer¬ 
cises  allow  ample  opportunities  for  collection  of  objective  data  on  unit 
performance.  In  March  1978,  ARI  produced  and  field  tested  at  Fort  Car- 
son,  Colo.,  a  low-cost  tactical  operations  measurement  system  that  is 
computer-supported  (Epstein,  1978).  The  system  includes  methods  and 
measurement  forms  for  recording  tactical  processes,  position  location 
data,  maneuver  routes,  and  tactical  outcomes  without  instrumented  data 
collection.  The  data  are  reduced  and  coded  using  machine-scorable  op¬ 
tical  scanning  forms  and  are  organized  by  the  computer  to  provide  a 
coherent  data  base  suitable  for  a  wide  variety  of  analyses.  The  NTC 
will  improve  this  data  collection  effort  by  using  instrumentation  that 
automatically  records  high-resc lution  position  location,  time,  and 
engagement  data. 

In  an  effort  to  implement  ES  in  the  field,  ARI  has  conducted  an 
ongoing  research  program  designed  to  incorporate  the  ES  methodology  into 
the  ARTEP  framework.  This  incorporation  may  not  be  a  trivial,  mechani¬ 
cal  problem.  ARTEP  currently  consists  almost  entirely  of  procedural 
statements  of  tasks,  conditions,  and  standards  for  one  side  in  a  two- 
sided  game;  that  is,  it  is  formatted  as  a  procedural  checklist.  ES, 
however,  was  designed  to  improve  training  by  increasing  realism,  particu¬ 
larly  with  respect  to  casualty  assessment.  The  problem  is  to  combine 
the  high-fidelity  outcome  data  available  with  ES  with  the  procedural 
formats  related  to  critical'  processes  included  in  ARTEP. 

The  modified  ARTEP  discussed  above  attempts  to  relate  the  high- 
fidelity  casualty  data  obtained  using  ES  methods  to  specific  ARTEP  proc¬ 
esses  and  more  global  task  statements.  For  example,  REALTRAIN  data 
from  the  combined  arms  test  at  Fort  Carson  are  being  analyzed  to  estab¬ 
lish  their  connection  with  ARTEP-type  task  statements.  What  has  evolved 
to  date  is  an  experimental  ARTEP  format  (Mirabella,  1977;  Sulzen  & 

Root,  1976)  including  mission-derived  tasks  and  conditions  supported 
by  process  and  product  measures  of  performance.  This  experimental  for¬ 
mat  provides  initial  attempts  to  reduce  limitations  on  performance 
measures  and  casualty  assessment  characteristic  of  the  original  ARTEP 
model . 

Although  the  modified  ARTEP  relates  objective  ES  outcome  data  to 
ARTEP  tasks  and  processes,  the  standards  problem  has  not  been  addressed 
except  in  a  norm-referenced  sense;  for  example,  what  is  the  performance 
level  of  Team  A  relative  to  some  earlier  point  in  training,  or  relative 
to  Team  B?  In  the  current  operational  system,  the  ARTEP  evaluators 
carry  around  in  their  heads  both  the  scales  of  performance  and  a  set 
of  performance  criteria  for  applying  these  scales.  Current  ARTEPs  do 
not  provide  an  appropriate  set  of  criteria.  More  often  than  not,  ARTEP 
standards  are  either  procedural  statements  or  undefined  cut  points; 
e.g.,  "casualties  shall  not  be  excessive."  Also,  the  current  standards 
are  not  suitable  for  two-sided  interactive  behavior.  With  regard  to 
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scales  of  performance,  current  ARTEPs  provide  little  if  any  guidance. 
ES  operationalizes  some  of  the  scales  of  performance,  although  it  does 
not,  in  and  of  itself,  provide  the  criteria  for  interpreting  the  per¬ 
formance  data,-  a  new  model  feature  is  required. 


Expected  Values  Subsystem 

The  expected  values  subsystem  of  the  COTEAM  evaluation  system  is 
designed  to  provide  performance  criteria,  standards,  or  benchmarks 
against  which  ES  field  exercise  outcomes  can  be  compared.  One  objec¬ 
tive  of  the  subsystem  is  to  provide  a  set  of  systematic  methods  for  de¬ 
fining  specific  behavioral  and  attrition  expected  values  for  ES  exer¬ 
cises  based  on  values  of  external  variables  in  a  specific  exercise  and 
the  assumption  of  combat-ready  opposing  forces.  The  evaluation  staff 
can  use  these  expected  values  in  making  evaluation  decisions  regarding 
unit  performance  and  in  training  diagnosis. 

To  systematize  and  standar’ize  the  evaluation  of  unit  performance 
in  ES  exercises,  performance  benchmarks  or  standards  must  be  defined. 

The  dynamics  of  two-sided  field  exercises  do  not  permit  exact,  deter¬ 
ministic  standards,  however.  The  standards  must  be  in  the  form  of 
probability  distributions,  tolerance  limits,  or  principle-derived  sets 
of  correct  solutions  to  tactical  problems.  To  establish  one  or  more 
of  these  types  of  standards,  large  amounts  of  ES  outcome  data  must  be 
obtained  so  that  the  characteristics  of  the  distributions  of  ES  outcomes 
can  be  determined.  Because  replication  of  field  exercises  is  difficult 
and  costly,  the  distribution  of  ES  outcomes  cannot  be  generated  in  field 
exercises.  As  part  of  the  COTEAM  research  program,  therefore,  inexpen¬ 
sive  simulations  of  ES  exercises  are  being  developed  to  provide  large 
amounts  of  valid  ES  outcome  data. 

The  concept  of  situation-specific  forecasting  is  being  used  to 
develop  an  ES  outcome  data  base  from  which  the  characteristics  of  the 
distributions  of  ES  outcomes  can  be  determined.  Situation-specific 
(Root  et  al.,  1978)  means  that  the  forecasting  procedures  replicate 
the  particular  field  exercise  conditions  as  closely  as  possible;  a 
forecasting  exercise  uses  the  same  force  ratios,  weapons  mix,  terrain, 
weather  conditions,  and  missions  as  the  corresponding  field  exercise. 
Given  a  valid  data  base  of  ES  outcomes,  expectations  about  tactical 
processes  and  casualties  can  be  derived  for  units  participating  in  ES 
field  exercises.  The  methods  being  developed  or  adapted  for  generating 
the  data  base  include  (a)  military  experts'  DELPHI,  (b)  board  war 
games,  and  (c)  computerized  ES  models. 


DELPHI 

The  DELPHI  technique  is  designed  to  exploit  and  enhance  experts' 
judgments  in  analysis,  evaluation,  and  forecasting.  In  its  simplest 
form,  the  DELPHI  technique  is  a  carefully  designed  series  of  individual 


interrogations  (usually  written  questionnaires)  coupled  with  informa¬ 
tion  and  opinion  feedback. 

For  example,  suppose  a  panel  of  military  experts  is  convened  to 
estimate  the  number  of  casualties  a  small  combat  unit  would  sustain 
during  a  specific  mission.  Each  panel  member  responds  individually 
with  an  initial  estimate.  The  person  in  charge  of  the  panel  collects 
the  data  and  computes  the  median  and  the  interquartile  range  (the  inter¬ 
val  containing  the  middle  50%  of  the  responses) .  Each  panelist  receives 
the  results  and  is  asked  to  make  a  new  estimate.  In  addition,  if  the 
panelists’  estimates  are  outside  the  interquartile  range,  panelists  are 
asked  to  indicate  why  their  responses  were  different  from  the  majority 
judgment.  Thus,  panelists  holding  extreme  views  are  forced  either  to 
defend  their  positions  with  explicit  reasons  or  to  move  toward  the 
majority  opinion.  Results  of  this  second  iteration  are  tabulated  and 
given  to  the  panelists  along  with  a  summary  of  reasons  presented  by 
those  who  hold  extreme  positions.  This  process  continues  for  four  or 
five  rounds  of  voting.  The  median  of  the  final  set  of  responses  is 
then  assumed  to  approximate  the  group  judgment,  and  the  range  of  re¬ 
sponses  may  be  presented,  representing  an  ordered,  weighted  series  of 
judgments,  i.e.,  a  set  of  answers  with  associated  priorities. 

A  variation  of  the  DELPHI  technique  that  may  have  particular  use 
in  COTEAM  combines  DELPHI  with  scaled  or  comparative  judgments  across 
a  set  of  stimuli.  (See  Wheaton  and  Mirabella,  1972,  for  an  illustra¬ 
tive  application  in  training  effectiveness  research.)  In  this  case, 
empirical  data  on  reference  training  units  might  be  used  as  anchors 
for  judgments  of  other  assumed  training  levels. 

The  DELPHI  technique  typically  leads  to  a  convergence  of  opinion. 
Other  benefits  gained  from  using  DELPHI  are  that  assumptions  are  made 
explicit,  a  broad  range  of  alternatives  is  produced,  and  judges  are 
sensitised  to  the  complexities  and  interactions  that  influence  alter¬ 
native  outcomes.  The  success  of  the  DELPHI  technique,  however,  depends 
on  several  factors — the  "expertness"  of  the  panelists,  the  type  and 
amount  of  feedback  provided,  and  the  quality  of  the  questionnaires 
used. 


A  pilot  study  to  determine  whether  military  personnel  could  make 
the  judgments  required  for  the  DELPHI  procedure  was  conducted  in  1977 
at  Fort  Ord,  Calif.,  as  part  of  a  REALTRAIN  rifle  squad  validation 
test  (Mirabella,  1977).  Exercise  participants  were  asked  how  they 
would  expect  infantry  squads  to  perform  on  a  number  of  variables  per¬ 
tinent  to  a  movement  to  contact  mission,  assuming  four  levels  of  train¬ 
ing:  BCT ;  Level  2  of  infantrv  \RTEP ;  Level  2,  plus  3  or  more  days  of 
SCOPES  training;  and  combat  cperienced.  Forecasts  were  made  for  a 
set  of  process  and  product  variables.  Results  suggested  that  military 
personnel  can  describe  differences  in  expected  performance  as  a  function 
of  hypothesized  level  of  training  and  that  expected  values  form  a  con¬ 
sistent  ordinal  scale  across  forecasters. 
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Board  Games 


A  second  method  of  producing  accurate  forecasts  was  explored  during 
the  armor  Combined  Arms  Test  (CATEST)  at  Fort  Carson,  Colo. ,  from  Janu¬ 
ary  to  April  1978  (Medlin,  1979a).  Using  scenarios  identical  to  those 
used  during  the  actual  field  exercises  (i.e. ,  force  mix,  OPORD,  weather, 
terrain,  etc.),  data  were  collected  using  the  Fort  Carson  Forecasting 
Game,  a  board  war  game.  The  game  wgp  developed  specifically  for  the 
Fort  Carson  exercises.  A  1:3,125  pictomap  of  the  exercise  lanes  served 
as  the  game  board.  A  hexagonal  grid  overlay,  with  each  hex  correspond¬ 
ing  to  50  m  of  terrain,  was  placed  over  the  map  to  help  standardize 
movement  rates,  detection  distances,  and  range  of  weapon  effectiveness. 

In  the  two-player  version  of  the  game,  each  player  maneuvers  his  forces 
on  a  separate  board,  out  of  sight  of  the  other  player.  Play  is  con¬ 
trolled  by  a  single  controller,  who  insures  observance  of  the  rules, 
keeps  exercise  time  moving,  delivers  indirect  fire,  provides  detections, 
and  assesses  casualties.  Although  the  controller  may  seem  to  have  con¬ 
siderable  power,  most  of  his  functions  are  clearly  and  precisely  expli¬ 
cated  in  the  rules.  The  only  subjective  decision  involves  detections; 
if  the  controller  is  unbiased  and  has  had  some  experience  with  the  actual 
exercise  terrain,  his  decisions  on  detections  can  be  fair,  accurate,  and 
easy  to  make. 

Results  of  the  study  indicate  a  few  small  differences  between  fore¬ 
cast  and  field  exercise  data;  in  general,  however,  the  data  from  the  two 
different  sources  are  quite  comparable.  Only  two  differences  of  any 
magnitude  were  encountered — the  greater  casualty-inflicting  ability  of 
the  tank  in  the  forecasting  exercises  and  the  greater  lethality  of  ar¬ 
tillery  in  the  field  exercises.  These  discrepancies  do  not  seem  to  be 
related  to  the  use  of  different  skills  in  the  field  and  on  the  game 
board.  Rather,  they  seem  to  be  due  mainly  to  slight  differences  in  the 
way  the  ES  casualty  rules  were  enforced  in  the  field  and  on  the  game 
board. 

A  second  study  was  conducted  to  evaluate  the  validity  of  board  war 
gaming  as  a  forecasting  technique  to  determine  performance  benchmarks 
(Medlin,  1979b) .  Using  data  from  the  Combined  Arms  Test  at  Fort  Carson 
for  March  1978  and  from  the  Fort  Carson  Forecasting  Game,  military 
judges  were  asked  to  distinguish  between  field  exercise  data  and  fore¬ 
cast  data.  Judgments  were  made  on  maps  of  exercise  maneuver  routes  and 
tables  summarizing  data  on  the  casualties  suffered  and  the  weapon  sys¬ 
tem  which  inflicted  each  casualty.  The  military  judges  were  not  able 
to  distinguish  between  field  and  forecast  maneuver  routes,  and  they 
tended  to  classify  forecast  casualty  data  as  field  exercise  casualty 
data.  Game  board  exercises,  therefore,  generate  data  similar  enough 
to  field  exercise  data  for  use  in  determining  characteristics  of  the 
distribution  of  ES  outcomes.  In  turn,  these  data  can  be  used  to  define 
performance  benchmarks  or  standards  for  units  participating  in  ES  field 
exercises . 
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Computer  Simulations 


Ultimately,  a  computer  simulation  can  be  used  to  generate  ES  out¬ 
come  data.  Input  parameters  can  be  varied  to  assess  the  effects  of 
external  conditions  and/or  unit  characteristics  on  field  exercises. 

To  develop  the  algorithms  necessary  for  a  computer  simulation  of  ES 
exercises,  the  nature  of  the  exercises,  the  processes  involved  in  reach¬ 
ing  the  final  outcomes,  and  the  relationships  among  the  variables  af¬ 
fecting  the  exercises  must  be  understood  much  better  than  at  present. 

An  exploratory  effort  at  building  a  computer  simulation  to  generate  a 
computerized  data  base  is  now  in  progress.  The  simulation  is  designed 
to  generate  hypothetical  casualty  effects -as  a  function'of  trarbnlrng 
level. 


Summary  of  Expected  Value  Generation 

All  of  the  above  methods  for  expectation  generation  may  be  thought 
of  as  combat  simulations,  or  theoretical  models  of  combat.  The  methods 
must  be  situation-specific,  replicating  the  field  exercise  battlefield 
as  closely  as  possible.  Convergent  and  empirical  validity  are  estab¬ 
lished  for  the  models  if  the  expectations  generated  by  them  agree  with 
each  other  and  with  the  observed  values  from  the  field  validation  exer¬ 
cises,  using  units  at  assumed  levels  of  proficiency  (for  example, 
"combat-ready") . 

If  the  expectations  and  observations  do  not  agrfee,  the  expectation 
generation  methodology  is  not  necessarily  deficient.  One  or  both  of 
the  field  exercise  units  may  not  be  at  the  assumed  level  of  training. 

This  confounding  of  possible  explanations  can  be  minimized  by  using  the 
leaders  of  the  field  exercise  units  to  generate  expectations  about  their 
own  field  exercises;  that  is,  by  having  the  same  leaders  forecast  out¬ 
comes  and  conduct  identical  field  exercises  in  which  outcomes  are 
recorded . 

Because  of  the  probabilistic  relationship  betv*;en  behavior  and  at¬ 
trition,  and  because  of  measurement  error,  discrepancies  between  expected 
and  observed  values  for  these  two  major  classes  of  combat  dependent  vari¬ 
ables  will  not  always  be  consistent  across  replications  of  the  same 
evaluation  exercise.  However,  large  divergence  from  expected  values 
for  either  class  of  variables  should  warrant  additional  substantiating 
analyses,  a  reassessment  of  the  expected-values  generation  methodology, 
or  additional  replications  of  the  evaluation  exercise.  The  value  of 
the  foregoing  approach  is  not  that  it  immediately  resolves  ambiguities 
associated  with  evaluating  combat  units;  rather,  the  value  of  the  ap¬ 
proach  is  that  it  provides  a  systematic  and  scientific  framework  link¬ 
ing  combat  theory  and  combat  unit  evaluation.  With  repeated  application, 
the  framework  will  help  resolve  ambiguities  associated  with  both  combat 
theory,  including  doctrine,  and  combat  unit  evaluation. 
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Comparison  Subsystem  for  Training  Diagnosis 


In  addition  to  the  diagnostics  listed  in  the  modified  ARTEF  and 
the  ARTEP  evaluators'  POI ,  the  COTEAM  system  introduces  new  techniques 
for  assessing  training  deficiencies  and  proficiencies.  The  approach 
follows  the  research  methodology  for  model  testing  and  building  out¬ 
lined  by  Rapoport  (1975).  After  methods  for  generating  expected  values 
have  been  validated,  the  expectations  are  used  as  "ideal,"  optimal,  or 
baseline  performance,  and  discrepancies  between  observed  and  expected 
performance  are  explored.  It  is  assumed  that  discrepancies  will  be 
neither  so  large  nor  so  unsystematic  as  to  make  the  comparison  meaning¬ 
less  or  useless.  If  systematic  discrepancies  are  found,  they  may  be 
interpreted  in  terms  of  unit  characteristics,  external  conditions,  doc¬ 
trine,  or  serendipitous  events.  If  proper  control  is  exercised,  the 
interpretations  may  rest  solely  on  unit  characteristics  and,  in  par¬ 
ticular,  on  the  training  level  of  the  unit.  If  the  unit  performs  better 
than  expected  on  some  task,  then  it  is  proficient  in  that  area.  If  the 
unit  performs  worse  than  expected,  it  lias  a  deficiency  that  can  be  ad¬ 
dressed  in  subsequent  training. 

In  the  initial  expectation  generation  process,  it  is  assumed  that 
both  units  are  combat-ready.  This  assumption  makes  it  possible  to  ob¬ 
tain  an  "ideal"  or  optimal  performance  reference  point.  For  example, 
suppose  that  a  reliable  discrepancy  has  been  observed  between  the  ex¬ 
pected  value  and  the  observed  value  for  attrition  in  a  two-sided  (Blue 
versus  Red) ,  free-play  simulated  combat  exercise.  What  inferences 
about  the  training  level  of  the  Blue  and  Red  forces  can  be  made?  Is 
Blue  attrition  lower  than  expected  because  Blue  training  is  suprastan- 
dard  or  because  Red  training  is  substandard  or  some  combination  of  the 
two? 


To  make  inferences  about  the  training  level  of  both  Blue  and  Red 
based  on  relative  attrition,  some  reference  points  or  baselines  are 
needed.  A  minimum  set  of  reference  points  are  expected  values  for  at¬ 
trition  under  the  following  conditions:  (a)  Blue  trained  but  Red  not 
trained,  (b)  Red  trained  but  Blue  not  trained,  and  (c)  neither  Blue 
nor  Red  trained.  These  additional  expected  values  for  attrition  under 
these  three  conditions  provide  anchor  points  on  a  scale  that  can  be 
used  to  interpret  observed  values  of  attrition  for  the  Blue  and  Red 
forces  and  to  relate  observed  attrition  to  training  level.  Additional 
reference  points  may  be  generated  by  representing  assumed  Blue  and  Red 
training  levels  with  other  than  extreme  parameter  values.  A  similar 
set  of  reference  points  may  be  generated  for  quantifiable  behavioral 
dimensions. 

Judging  unit  performance  on  the  basis  of  a  comparison  between 
what  is  expected  of  a  unit  and  what  it  actually  does  is  an  attractively 
simple  idea.  The  comparison  provides  the  framework  needed  to  evaluate 
units  in  a  context  broader  than  a  specific  exercise  or  series  of  exer¬ 
cises,  yet  it  can  be  made  sensitive  to  the  particular  conditions  under 
which  a  unit  is  observed.  Depending  on  the  purpose  of  a  particular 
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evaluation  exercise,  the  comparison  process  can  address  specific  train¬ 
ing  diagnostic  questions,  provide  data  for  training  management  com¬ 
parable  to  qrade-equivalent  scores,  answer  doctrinal  questions,  or 
indicate  readiness  for  combat. 


SUMMARY 

The  Combat  Operations  Training  Effectiveness  Analysis  Model  (COTEAM) 
is  an  evaluation  system  for  combat  unit  performance  that  systematically 
addresses  the  weaknesses  inherent  in  the  current  ARTEP.  ARI  developed 
the  model  and  is  conducting  research  on  all  aspects  of  the  system.  Using 
the  ARTEP  as  a  starting  point,  the  COTEAM  model  proposes  a  modified  ARTEP 
manual  that  includes  task/subtask  diagnostics  that  facilitate  the  evalua¬ 
tion  and  training  feedback  processes  and  an  evaluator  program  of  instruc¬ 
tion  (POI)  that  teaches  evaluators  roles,  duties,  and  how  to  conduct  a 
field  exercise.  Furthermore,  the  task/subtasks  listed  in  the  manual  are 
being  scientifically  and  systematically  analyzed  to  determine  the  criti¬ 
cal  events  that  must  occur  for  a  unit  to  perform  its  mission. 

The  rest  of  the  COTEAM  evaluation  system  is  driven  by  the  modified 
ARTEP  manual.  The  evaluation  environment  is  defined  by  the  modified 
ARTEP  manual  (mission,  posture,  personnel,  weapons,  etc.)  and  the  field 
conditions  (weather,  terrain,  time  of  day,  etc.).  The  unit  to  be  evalu¬ 
ated  enters  this  environment  with  certain  unit  characteristics  (indi¬ 
vidual  skills,  unit  training  level,  leadership,  etc.).  A  realistic  com¬ 
bat  environment  should  be  provided  to  assess  the  combat  unit's  perfor¬ 
mance  on  the  ARTEP  tasks/subtasks.  Engagement  simulation  (ES)  is  a 
two-sided,  free-play,  tactical  maneuver/fie Id  exercise  that  allows  im¬ 
mediate  and  realistic  casualty  assessment.  ES  field  exercises  provide 
a  realistic  combat  environment  in  which  combat  units  can  be  evaluated. 
(Although  a  high-fidelity  combat  environment  maximizes  the  usefulness 
of  COTEAM,  the  model  can  be  applied  where  high-fidelity  technology  is 
not  available  if  suitable  methods  are  used  to  generate  expectations.) 

The  expected  values  subsystem  of  COTEAM  is  a  means  of  evaluating 
performance  in  tactical  field  exercises.  Given  the  external  conditions 
as  defined  by  the  ARTEP  manual,  the  field  environment,  and  the  ES  ex¬ 
ercise  rules,  expected  values  or  expectations  are  generated  for  the 
process  and  product  outcomes  of  the  field  exercises.  These  expected 
values  provide  a  baseline,  benchmark,  or  standard  with  which  observed 
unit  performance  can  be  compared.  Considerable  research  has  been  di¬ 
rected  at  the  expected  values  subsystem  of  the  COTEAM  model.  The  DELPHI 
technique,  board  war  gaming,  and  computer  simulations  are  being  devel¬ 
oped  as  methods  of  generating  expectations.  Although  the  feasibility 
of  the  DELPHI  technique  and  computer  simulations  is  still  being  explored, 
board  war  gaming  has  been  established  as  a  means  of  generating  expected 
values  for  process  and  product  outcomes  extremely  similar  to  those  ob¬ 
served  in  field  exercises. 
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A  comparison  of  the  expected  and  observed  outcomes  facilitates 
evaluation  of  unit  performance  and  decisions  concerning  training  level 
and/or  combat  readiness.  This  comparison  process,  in  conjunction  with 
the  ARTEP  manual  diagnostics  and  the  evaluator  training  provided  in 
the  POI,  provides  information  for  feedback  to  the  unit,  particularly 
feedback  concerning  training  deficiencies  and  proficiencies.  This  feed¬ 
back  can  be  used  to  structure  the  unit's  training  programs. 

The  COTEAM  evaluation  system  is  one  of  the  models  being  developed 
as  part  of  ARI's  research  on  unit  evaluation.  The  system  introduces  a 
realistic  simulated  combat  environment  in  which  units  perform  tactical 
operations  and  in  which  objective  data  can  be  obtained,  specific  pro¬ 
cedures  for  determining  standards  against  which  unit  performance  can  be 
compared,  and  techniques  for  assessing  training  level,  combat  readiness, 
and  training  deficiencies  can  be  refined.  Research  is  being  conducted 
on  all  aspects  of  the  model,  and  modifications  to  COTEAM  will  incorpo¬ 
rate  the  results  of  these  efforts.  The  ultimate  goal  of  this  research 
is  to  develop  a  criterion-referenced  system  for  evaluating  unit  tactical 
performance. 
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1  USA  Recruiting  Cmd,  Ft  Sheridan.  ATTN:  USARCPM  P 
1  Senior  Army  Adv.,  USAFAGOD/TAC,  Elgin  AF  Aux  Fid  No  9 
1  HQ,  USARPAC,  DCSPER.  APO  SF  96558.  ATTN:  GPPE-SE 
I  Stimson  Lib,  Academy  of  Health  Sciences,  Ft  Sam  Houston 
1  Marine  Corps  Inst .  ATTN:  Deen-MCI 
1  HQ.  USMC.  Commandant.  ATTN:  Code  MTMT 

1  HQ.  USMC.  Commandant.  ATTN  Code  MPI-20-28 

2  USCG  Academy.  New  London,  ATTN:  Admission 
2  USCG  Academy,  New  London,  ATTN:  Library 

1  USCG  Training  Ctr.  NY,  ATTN:  CO 
1  USCG  Training  Ctr.  NY.  ATTN  Educ  Svc  Ofc 
1  USCG,  Psychol  Res  Br.  DC.  ATTN  GP  1/62 
1  HQ  Mid  Range  Br.  MC  Dat.  Quantico.  ATTN  PAS  Div 
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I  US  Manna  Corps  Liaison  Ofc,  AMC.  Alexandria.  ATTN  AMCGS  • 

I  USATRAOOC.  Ft  Mmuor  ATTN  AFRO  ID 
6  USATRADOC.  Ft  Monroe.  ATTN  AIPR  AO 
?  USATRADOC.  Ft  Monroe.  ATTN  AT7S  FA 
1  USA  f  ones  Cmd.  F t  M»  Pheison,  A II  N  l  ibrary 
?  USA  Aviation  Test  B*I.  Ft  Rucker.  AT  IN  STFBG  PO 
1  USA  Agi  y  for  Aviation  Safety  t  i  Rucker.  A I  IN  Liltfary 
I  USA  Agry  to i  Aviation  S  afety.  Ft  Rutkw,  A I  IN  Educ  Advisor 
I  USA  Aviation  Sch.  Ft  Hurkm,  ATTN  PO  Drawer  O 
I  HQUSA  Aviation  Sys  Cmd.  St  Louis.  ATTN  AMSAV  7DR 
?  USA  Aviation  Sys  Test  A*  t  .  I  clw.mJs  Af  R.  A!  TN  SAV  II  I 
I  USA  An  Or d  Srh.  Ft  Bln*.  A1TN  A  TSA  TFM 

l  USA  An  Mobility  RmIi  A  Oi  v  Lal».  Moffett  f  lit.  AI  TN:  SAVOl  AS 
I  USA  Aviation  Sett  Res  Trig  Mqt.  Ft  Rucker.  ATTN  ATST  T  RlM 
1  USA  Aviation  Sett.  CO.  Ft  Rucker  ATTN  ATST  D  A 
t  HO  OARCOM.  Alexandria.  ATTN  AMXCD  TL 
I  HQ.  OARCOM  Alexandria.  ATTN  COR 
1  US  Military  Academy.  West  Point,  ATTN  Serials  Unit 
1  US  Militaiy  Academy  West  Point.  ATTN  Ofc  of  Milt  Ldishp 
1  US  Militaiy  Academy  West  Point  ATTN  MAOR 
1  USA  Standardisation  Gp  UK.  FPO  NY.  ATTN  MASF  (»C 
1  Ole  of  Naval  Rscft.  Aifnujtoo.  ATTN  Code  45? 

J  Ofc  of  Naval  Rscft  Arlington.  ATTN  Code  458 
I  Ofc  of  Naval  Rscft.  Arlington.  ATTN  Code  450 
1  Ofc  of  Naval  Rsch.  Arlington.  ATTN  Code  441 
1  Naval  Aemspc  Med  Res  (  ah  Pensacola.  ATTN  Acout  Sch  Oiv 
l  Naval  Aeiospc  Med  Res  Lah.  Pensacola.  ATTN  Code  151 
1  N.ival  Aeiospc  Med  Res  Lab.  Pensacola  ATTN:  Code  L5 
1  Chief  of  NavPers,  ATTN  Peis  OR 
1  NAVAIRSTA.  Norfolk.  ATTN  Safety  Ctr 
I  Nav  Oceanographic.  PC  ATTN  Code  6251,  Charts  A  Tech 
1  Center  of  Naval  Anal.  ATTN  Dim- Ctr 
1  Nav  An  SysCont.  ATTN  AIR  5313C 
I  Nav  BuMed.  ATTN  713 
1  NavHellcopterSuhSqua  ?.  FPOSF  96601 
1  AFMRL  (FT)  Williams  AF8 
1  AFHRL  ITT)  Lowry  AFB 
1  AFHRL  (AS)  WPAFB.  OH 
?  AFHRL  (DOJZ)  Brooks  AFB 
1  AFHRL  (DOJN)  l  ackland  AFB 
1  HOUSAF  (INYSD) 

1  HQUSAF  ( DP  X  X  A ) 

1  AFVTG  IRP)  Randolph  AFB 

3  AMRL  (HE)  WPAFB  OH 

?  AF  Inst  of  Tech  WPAFB.  OH  ATTN  ENE^SL 
t  ATC  (XPTD)  Randolph  AFB 

1  USAF  AeroMert  l  ib.  Brooks  AFB  (SUL  4).  ATTN  DOC  SEC 
1  AFOSR  (Nl ).  Ailmgton 

1  AF  Log  Cmd.  McClellan  AFB.  ATTN  ALC/DPCRB 

1  Air  Force  Academy.  CO.  ATTN;  Dept  of  Bel  Sen 
5  NavPers  A  Dev  Ctr  Sait  Diego 

2  Navy  Med  Nemopsychiatnc  Rsch  Unit.  San  Diego 
1  Nav  Electronic  Lah.  San  Diego.  ATTN  Res  Lah 

1  Nav  TntgCen.  San  Diego.  ATTN  Code  9000  Lilt 
1  NavPostGraSch,  Monteiey,  ATTN.  Code  55Aa 
1  NavPostGraSch.  Monterey.  ATTN  Code  2124 
1  Nav TmgEquipCtr,  Orlando,  ATTN  Tech  Lib 
1  US  Dept  of  Labor,  DC.  ATTN  Manpower  Admin 
1  US  Dept  of  Justice.  DC,  ATTN:  Drug  Enforce  Admin 
1  Nat  Bur  of  Standards.  DC.  ATTN.  Computer  Info  Section 
I  Nat  Clearing  House  foi  MH  Info,  Rockville 
I  Denver  Federal  Ctr.  I  akewood,  ATTN  BLM 
I?  Defense  Documentation  Center 

4  Dir  Psych.  Army  Hq.  Russell  Ofcs,  Canberra 

I  Scientific  Advsr,  Mil  Bd.  Army  Hq.  Russell  Ofc.s,  Canberra 
I  Mil  and  Air  Attache.  Austrian  Embassy 

1  Centie  de  Recherche  Des  Facteurs,  Humame  de  la  Defense 
Natmnale.  Brussels 

2  Canadian  Joint  Staff  Washington 

I  C/Air  Staff.  Royal  Canadian  AF.  ATTN  Pars  Std  Anal  Hr 

3  Chief.  Canadian  Def  Rsch  Staff.  ATTN  C/CRDS(W) 

4  British  Def  Staff  British  Embassy.  Washington 


1  Dr  f  A  Civil  Inst  of  Enviro  Medicine.  Canaria 
I  AIR  CRESS.  Kensington  ATTN  Info  Sys  B» 

I  Mililartrpsykologisk  1  lenestc  .Copenhagen 
1  Military  Attache,  F tench  Fmfiassv  AT  TN  Doc  Ser 
I  Medei  m  Cf*et.  C  f  R  P  A  Arsenal.  Toulon  Naval  f  lance 
1  Pnn  Scientific  Off.  A|*pl  Hum  (  ogr  Rsch  Oiv  Mmistiy 
of  Defense .  New  Delhi 

I  Peis  Rsch  Ofc  Lilnaiy  AKA,  Isiael  Defense  I  nice* 

I  Minister  is  van  Defensie.  DOOP/K l  Afd  Sociaal 
Ptychologitche  7aken.  The  Hague  Nethn lands 
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